Information maximization in a network of linear neurons

It is known since the work of Hubel and Wiesel [3], that many cells in the early visual areas of mammals have characteristic response properties. For example, there are cells sensitive to lightdark contrast and cells sensitive to edges of a certain orientation. In a series of articles [4, 5, 6], Linkser showed experimentally that similar response properties can be developed in a simple feed-forward network of linear neurons by using a Hebbian learning rule [2]. The structures that emerged on the different layers of Linsker’s network were spatial-opponent (also called center-surround) cells, orientation-selective cells, and orientation columns. Interestingly, these structures emerged without any structured input to the network. This is an important aspect because cells with such characteristics have been found in monkeys even before birth, i.e., before they could have experienced any visual input. Linsker showed in [7, 8, 9] that the self-organization process leading to these receptive field structures is consistent with the principle of maximum information preservation, or “infomax” principle for short. In the context of neural networks, the infomax principle states that the transfer function from one neural layer to the next should preserve as much information as possible. If we call the state of the first layer the stimulus and the state of the second layer the response, then the infomax principle states that the response should contain as much information about the stimulus as possible. This text is intended as an overview of a part of Linsker’s work. The analysis makes use of some concepts from information theory. The book by Cover and Thomas [1] is an extensive (and expensive) reference on that subject; Shannon’s original paper [10] is also a very good reading (and is freely available).